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ABSTRACT 



In an effort to determine differences between 



students who respond to surveys and those who do not, New Mexico 
State University at Alamogordo (NMSU-A) undertook a study of students 
in fall 1992. Students were asked to complete an 18-item 
questionnaire while waiting in lines during registration. The 1,618 
out of 2,114 (76.57.) students who completed the survey, completed 
registration, and attended classes were compared to the 496 (23.5%) 
of students who enrolled, attended classes but did not complete the 
survey. The study used data from the NMSU-A Student Information Files 
to determine characteristics of the two groups. Study results 
included the following: (1) while students in the 16-22 and 23~29 age 
groups were more likely not to complete Lhe survey than was expected, 
the difference between other age groups was not significant; (2) no 
significant differences were found in ethnicity or enrollment status 
(i.e.,' continuing, readmitted, first-time, or transfer students) 
between responders and non-responders ; (3) females were more likely 
to complete the survey than males; (4) full-time students and 
students taking between 6 and 12 credit hours were more likely to 
complete the survey; and (5) completers had a slightly higher grade 
point average (3.01) than those who did not (2.97). Study results 
suggest that students who respond to surveys may differ from those 
who do not and that researchers should exercise caution. (Contains 23 
references.) (Twenty-two tables and the survey instrument are 
appended.) (KP) 
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Are Students Who Complete Questionnaires 
Different from Students Who Don't? 

Abstract 

Volunteer bias in student-generated data may affect validity. Accountability mandates 
require that higher education institutions report student-oriented data. Because of legal and 
ethical considerations, in most cases, students cannot be required to provide these data. Faced 
with this dilemma and the need to assess student attainment, persistence, and completion rates, 
researchers often must supplement student databases with voluntary student responses. This 
study suggests that students who completed survey questionnaires were different from students 
who did not fill out the same survey. Literature reviewed for this study examined studies that 
considered mail, telephone, and other survey research. 

Description of Study 
Random selection of subjects will lead to the most valid results. A recent opinion 
expressed in the Chronicle of Higher Education (Tanur, 1994) stated "if sampling is not done at 
random, the results of the survey will give a biased picture of the population." The article further 
suggests that "the decision about who should participate in the survey also should not be left to 
respondents." However, institutional researchers often cannot rely on randomly selected subjects 
and cannot require students to provide information. Problems may be caused for the researcher 
when students do not respond. "In survey research, nonresponse has been one of the major 
concerns about survey accuracy among social researchers, students of public opinion, and media 
practitioners" (American Statistical Association, 1974; Cranberg, 1975; Chang, Voelker, & Lee, 



Student Questionnaires 
3 



1989; Tanur, 1994). These concerns are not new. Reuss, in a 1943 study of college students 
attempted to answer these questions: '"Who responds to a questionnaire? Are the persons 
responding different from those not responding? In what respects do they differ from the non- 
response group?'"(Reuss, 1943, p. 433). This study attempts to answer these same questions. 

When a survey "fails to achieve a high response rate, there is some question about a 
possible bias due to nonresponse by part of the population" (Boser, 1988, p. 1). Smith (1983) 
suggests that "nonresponse can seriously bias survey estimates and distort inferences." It has 
been shown that "questionnaire-return bias is pervasive" (Schwirian & Blaine, 1966-67, p. 663). 
This is likely to be a problem with surveys because even under optimum conditions it "is almost 
impossible to obtain answers from every person in a large sample" (Rosnow & Rosenthal, 1984, 
p. 90). For telephone surveys, "variations in levels of response can affect the existence and 
magnitude of relationships observed in the resulting data" (Traugott, Groves & Lepkowski, 
1987). If the number of nonrespondents is more than 20 percent, "it is very likely that most of the 
findings ... could have been altered considerably" (Borg& Gall, 1989, p. 443). Even a typical 
response rate of 75% (Tanur, 1994) may still allow volunteer bias to creep into the study results. 
It has been suggested "that there is no simple, general, accurate way of measuring nonresponse 
bias" (Smith, 1983, p. 402). 

Where does this leave the institutional researcher who must rely on "voluntary" 
information provided by students on survey questionnaires? Considerable research has been done 
to better understand the "volunteer subject". However, most of this research has been done in 
fields of behavioral research and public opinion polling. We need to try to better understand how 



ERIC 



4 



Student Questionnaires 
4 



college students who complete student surveys are different from those students who do not fill 
out the surveys. This may be critical because colleges and universities have Mttle choice but to 
rely on data volunteered by students. This is true for a variety of reasons including a student's 
right to privacy and ethical considerations related to informed consent Colleges must provide an 
escalating amount of information to SPRE's, accreditors, state and federal governmental agencies, 
and lawmakers. These entities want to know more and more about our students. In many cases, 
the very agencies that want us to provide information are the same ones that restrict our ability to 
collect information. Clearly the response rate of "voluntary" questionnaires is different from 
"mandatory" questionnaires. When given the choice, some students will choose not to fill out a 
voluntary questionnaire. Behavioral research suggests that volunteer subjects are different from 
those that do not volunteer (Rosenthal, 1965; Rosenthal & Rosnow, 1969). Cranberg (1975) 
suggests that "non-response error would be possible in any mail survey that seeks information that 
could reflect unfavorably on prospective respondents." Donald (1960) concluded that "the higher 
the involvement, in terms of active participation, knowledge and understanding of the 
organization, and loyalty to it, the fewer the stimuli required to induce a response." It is possible 
to conclude that engaged college students are more likely to complete surveys than nonengaged 
students. The possible implications should not be ignored. If this is also true of survey 
respondents, then can the results of data generated solely by volunteers be applied to 
nonvolunteers? Some researchers have "argued that because there are many reasons why a 
person might or might not volunteer to participate in behavioral research, volunteer status is . . . 
'of little interest'" (Kruglanski, 1973; Rosenthal & Rosnow, 1975). But researchers should be 
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interested if it is true that "implications for data interpretation where mail questionnaire response 
is incomplete were shown to be especially strong for attitudinal data concerning estimates of the 
specific behavior of others and of own attitudes, understanding, and future behavior" (Donald, 
1960, p. 113). These are the types of research questions we often ask. How much can we rely on 
the answers we get from students? 

Colleges should not collect data that cannot be used to improve student learning or 
institutional effectiveness. Once collected, data should only be used in appropriate ethical ways. 
The institutional researcher and college decision makers must know if the results of survey 
research are valid for students. Literature reviewed suggested possible differences between 
respondents and nonrespondents in age, ethnicity, gender, and education and intelligence. 
Age. 

Nonrespondents are older (Hochstim & Athanasopoulos, 1970). Hogan found that a 
"traditional-aged student (under 25 years old)" was more likely to participate in surveys. A study 
of community college students found the "responder group being older than the non-rcsponder 
group by about 1 year" (Carifio, Biron, & Shwedel, 1987). "Participants in survey research 
studies tend to be younger than non-participants" (Rosenthal, 1965, p. 395). 
Ethnicity 

Ethnicity differences between respondents and nonrespondents are inconclusive. One 
study found that nonrespondents are more likely to be white (Hochstim & Athanasopoulos, 
1970). However, Hogan (1985) found that white students were more likely to respond. Another 
study found no significant difference (Carifio, Biron, & Shwedel, 1987). 
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Gender 

Nonrespondents are more likely to be male (Gannon, Nothem & Carroll, 1971; Hochstim 
& Athanasopoulos, 1970; Nielsen, Moos & Lee, (1978). "Females were more likely to respond 
than were male graduates" (Hogan, 1985, p. 19). Carifio, Biron, & Shwedel (1987) found no 
significant differences. Rosenthal (1965) reports mixed results in gender differences of volunteer 
and non-volunteer subjects. 
F/lnration and Intelligence 

Analysis of high school rank, freshman class rank, and first semester GPA revealed that 
"persons responding to the questionnaire were superior in intelligence to those not responding" 
(Reuss, 1943, p. 433). Nonrespondents had significandy lower freshmen GPAs (Nielsen, Moos & 
Lee, 1978). Nonrespondents are likely to have lower levels of education (Gannon, Nothern & 
Carroll, 1971). Nonrespondents have lower mean IQ's (Macek & Miles, 1975). Students who 
"refused to participate in the survey are precisely those who have the lowest potential for doing 
college-level work" (Ellis, Endo & Armer, 1970, p. 107). The mean GPA of survey participants 
was "consistently higher than that earned by the population as a whole" but the differences were 
not statistically significant (Hogan, 1985). "Overall, the evidence suggests that, in comparison 
with non-volunteers, volunteers are likely to be brighter" (Rosenthal, 1965, p. 396). 

This paper attempts to determine how student respondents are different from 
nonrespondents. It will include a review of research findings about volunteer bias and differences 
found between respondent and nonrespondent groups. 
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Method 

This paper is based on research conducted at a small community college. Students were 
asked to complete an 18-item survey questionnaire in Fall 1992 [see Appendix]. Students were 
handed survey questionnaires while they were waiting in tine during registration. They were not 
"required" to complete the survey. However, unless they asked, students were not "informed" 
that they were "not required" to fill out the survey questionnaire. A total of 1618 of 21 14 
(76.5%) students completed the survey, completed the registration process, and attended classes. 
These students are designated in the tables as "SURVEY". Just less than a quarter of the students 
enrolled, attended classes, but did not complete the survey, 496 of 21 14 (23.5%). The students 
are designated in the tables as "NOSURVEY". Chi square analysis was used to determine if these 
groups were significantly different (p < .01). 

This study also examined how students who completed the survey and those who did not 
differed in terms of completion. Several different student cohorts were considered. These 
included all students who attended in the Fall of 1992 and students who enrolled at NMSU-A for 
the first time during the Fall 1992 semester. The purpose was to attempt to better understand 
their enrollment profile. A student database was established using data from the NMSU-Las 
Cruces Student Information Files (SIF). An extract of that file was created that contained only 
those students who were still enrolled on the census date and only enrolled at NMSU-A. The 
cohort of first-time students had an enrollment status that was coded as "First Time Freshmen" 
(13) as defined by the New State University Student Information Files Data Element Dictionary, 
1992-1993. 
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The objective of the study was to find out the enrollment status of these students. 
Students were tracked for the following semesters: Fall 92, Spring 93, Fall 93, Spring 94, and 
Fall 94. Students that were enrolled during any of these study semesters were considered as 
persisters. The persisters are displayed on the report as "Persisters" While some students 
dropped out and did not return, many students "stopped out" sometime during the study period, 
but they enrolled during a subsequent semester. These students are displayed on the report as 
"Stopouts". Some students decided to transfer to other colleges at some point during the study 
period. For this study, the only students that were listed in ihis category were those that 
transferred to NMSU-Las Cruces and whose names appeared on the Branch to Main Campus 
Report, that is prepared every semester by the Office of the Dean, NMSU College of Human and 
Community Services. These students are displayed on the report as transfers to NMSU". 

Students who were awarded a certificate or a degree by NMSU-A were displayed on the 
report as "Certificates" or "Degrees". This information was provided by NMSU-A Student 
Services. The Cumulative Total Completion Rate (Completers) is the cumulative total 
students who either obtain a certificate, degree, or transfer to NMSU within the study period 
divided by Total Students in Cohort (This may not equal the sum of Total Certificates, Total 
Degrees, and Total Transfers to NMSU because only the first occurrence is counted — in other 
words, while a student may get a certificate, a degree, and then transfer - that student may only 
"complete" once). 
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Results 

Findings suggest that these groups differed in the number of credit hours taken and their 
gender. They did not differ in ethnicity, age, or enrollment status. The analysis of CPAs and and 
student completion was inconclusive. 

Students in the two younger age categories, 16-22 and 23-29 were more likely not to 
complete the survey than was expected (See Table 1). However, this difference was not 
statistically significant. The mean age for students who did not complete the survey was 31.66 
years. The mean age for students that did complete the survey was 30.39 years. 

Table 2 shows that there was no significant difference in ethnicity between those that 
completed the survey and those that did not. However, there were some interesting differences 
among the minority groups. 

Table 3 shows that females were more likely to complete the survey than males. This 
result is consistent with most studies found in the literature. 

Table 4 shows that there was no significant difference in enrollment status between those 
that completed the survey and those that did not complete the survey. Enrollment status includes 
continuing students, readmitted students, first-time students, and transfer students. 

Table 5 shows that fulltime students and students that took between six and 12 credit 
hours were more likely to complete the survey than were students who took fewer than six credit 
hours. 

Table 6 shows how the students who completed the survey and those who did not 
complete the survey compared when their cumulative grade point average (GPA) was examined. 
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Various categories were calculated. The table shows that the differences between the two groups 
were not extreme. Overall, students who completed the survey had a slightly higher GPA (3.01) 
than those who did not complete the survey (2.97). However, students who did nol fill out the 
survey did have higher GPAs in the following categories: females who took less than six credit 
hours in Fall 1992 (2.88/2.81); females who took six to eleven credit hours during Fall 1992 
(3.09/3.07); males who took six to eleven credit hours during Fall 1992 (2.99/2.98); and males 
who were enrolled fulltime during Fall 1992 (3.30/3.1 1). 

Table 7 shows how the number and rate of completers differ between the two groups. 
When all students were considered, students who completed the survey (22%) had a higher 
completion rate than students who did not complete the survey (20%). Students who took fewer 
than six credit hours during Fall 1992 were more likely to complete if they filled out the survey 
(8%) than students who did not fill out the survey (6%). Conversely, students who did not 
complete the survey were more likely to be completers if they took six to eleven credit hours or if 
they were fulltime students. Complete results are reported in the Appendix. 

Table 8 presents how the cumulative completion rate for first-time students who 
completed the survey differed from first-time students who did not complete the survey. Students 
in every category reported who completed the survey were more likely to complete than students 
who did not complete the survey. Complete results are reported in the Appendix. 

Discussion 

The results of this study suggest that researchers who rely on volunteered data supplied by 
students who complete surveys should be cautious. This limited study suggests that students 
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who complete surveys may be different from students who do not complete the survey. Future 
research includes, testing subsequent student survey results to see if differences found in this 
study are substantiated. 
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Table 1 

Students Who Ccmpleted and Did Not Complete Survey 

by Age 

Students Who Attended - Fall 1992 
New Mexico State University - Alamogordo 



HROITP 

VJlvv/ VJ I 

Frequency 


AGE 


Expected Percent 
Row Pet 
Col Pet 


16-22 


23-29 


30-49 


50 + 


Total 


Completed Survey 


440 
427.08 
20.81 
27.19 
78.85 


441 

429.37 
20.86 
27.26 
78.61 


652 
666.64 
30.84 
40.30 
74.86 


85 
94.91 
4.02 
5.25 
68.55 


1618 
76.54 


Did not Complete 
Survey 


118 

130.92 
5.58 
23.79 
21.15 


120 
131.63 
5.68 
24.19 
21.39 


219 
204.36 
10.36 
44.15 
24.14 


39 
29.10 
1.84 
7.86 
31.45 


496 
23.46 


Total 


558 


561 


871 


124 


2114 




26.40 


26,54 


41.20 


5.87 


100.00 



DF Value Prob 
Chi-Square 3 8.785 0.032 



Effective Sample Size = 21 14 
Frequency Missing = 143 
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Table 2 

Students Who Completed and Did Not Complete Survey 

by Ethnicity 
Students Who Attended - Fall 1992 



GROUP 






ETHNICITY 






Frequency 














Expected Percent 
Row Pet 
Col Pet 


Hispanic 


Black 


Native 
Indian 
People 


Asian 


Anglo 
Other 


Total 


Completed Survey 


287 
283.19 
13.58 
17.74 
77.57 


87 
92.61 
4.12 
5.38 
71.90 


77 
78.83 
3.64 
4.76 
74.76 


20 
21.43 
.95 
1.24 
71.43 


1147 
1141.9 
54.26 
70.89 
76.88 


1618 
76.54 


Did not Complete 
Survey 


83 
86.81 
3.93 
16.73 
22.43 


34 
28.39 
1.61 
6.85 
28.10 


26 
24.17 
1.23 
5.24 
25.24 


8 

6.57 
.38 
1.61 
28.57 


345 
350.06 
16.32 
69.56 
23.12 


496 
23.46 


Total 


370 


121 


103 


28 


1492 


2114 




17.50 


5.72 


4.87 


1.32 


70.58 


100.0 



Chi-Square 



DF Value 
4 2.352 



Prob 
0.671 



Effective Sample Size « 21 14 
Frequency Missing = 143 
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Table 3 

Students Who Completed and Did Not Complete Survey 

by Gender 
Students Who Attended - Fall 1992 



GROUP . 




GENDER 




Frequency 








expected Jrerceni 
Row Pet 


Female 


Male 


Total 


1 Col Pet 








Completed Survey 


1004 
972.02 


614 
645.98 


1618 




47.49 


29.04 


76.54 




62.05 


37.95 






79.06 


72.75 




Did not Complete 
Survey 


266 
297.98 
12.58 


230 
198.02 
10.88 


496 
23.46 




53.63 


46.37 






20.94 


27.25 




Total 


1270 


844 


2114 




60.08 


39.92 


100.00 



Chi-Square 



DF Value Prob 
1 11.229 0.001 



Effective Sample Size = 2114 
Frequency Missing = 143 
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Table 4 

Students Who Completed and Did Not Complete Survey 

by Enrollment Status 
Students Who Attended - Fall 1992 
New Mexico State University - Alamogordo 



GROUP 
Frequency 



ENROLLMENT STATUS 



Expected Percent 
Row Pet 


Continuing 


Readmitted 


First-time 


Transfer 


Total 


Col Pet 












Completed Survey 


857 
851.10 


267 
270.18 


300 
290.84 


194 
205.89 


1618 




40.54 


12.63 


14.19 


9.18 


76.54 




52.97 


16.50 


18.54 


11.99 






77.07 


75.64 


78.95 


72.12 




Did not Complete 
Survey 


255 
260.90 
12.06 


86 
82.82 
4.07 


80 
89.16 
3.78 


75 
63.11 
3.55 


496 
23.46 




51.41 


17.34 


16.13 


15.12 






22.93 


24.36 


21.05 


27.88 




Total 


1112 


353 


380 


269 


2114 




52.60 


16.70 


17.98 


12.72 


100.00 


DF Value 


Prob 








Chi-Square 


3 4.487 


0.213 









Effective Sample Size = 2114 
Frequency Missing = 143 
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Table 5 

Students Who Completed and Did Not Complete Survey by 

Number of Credits Taken 
Students Who Attended - Fall 1992 
New Mexico State University - Alamogordo 



GROUP 
Frequency 



CREDITS TAKEN 



Expected Percent 
Row Pet 
Col Pet 


FULLTIME 


FEWER 
THAN 6 
CREDITS 


6T011 
CREDITS 


Total 


Completed Survey 


407 
376.56 
19.25 
25.15 
82.72 


631 
690.37 
29.85 
39.00 
69.96 


580 
551.07 
27.44 
35.85 
80.56 


1618 1 
76.54 


Did not Complete 
Survey 


85 
115.44 
4.02 
17.14 
17.28 


271 
211.63 
12.82 
54.64 
30.04 


140 
168.93 
6.62 
28.23 
19.44 


496 1 
23.46 


Total 


492 


902 


720 


2114 




23.27 


42.67 


34.06 


100.00 



Chi-Square 



DF Value Prob 
2 38.717 0.000 



Effective Sample Size = 21 14 
Frequency Missing = 143 
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Table 6 

Cumulative Grade Point Average and Gender 
Students Who Attended - Fall 1992 



m ■ 1 

\ 


Completed Survey 


Did Not Compete 
Survey 


• 






;., • . . • . ' - ' :: •' ; 


n 


Cumulative 
Grade Point 
Average 


n 


Cumulative 
Grade Point 
Average 


All Students 


1618 


3.01 


496 


2.97 


Females 


1004 


3.04 


266 


2.98 


Males 


614 


2.97 


230 


2.96 


Females who took fewer than 6 
credits 


368 


2.81 


133 


2.88 


Males who took fewer than 6 credits 


263 


2.89 


138 


2.85 


Females who took 6 to 1 1 credits 


361 


3.07 


83 


3.09 


Males who took 6 to 1 1 credits 


219 


2.98 


57 


2.99 


Females - Fulltime 


275 


3.30 


50 


3.07 


Males - Fulltime 


132 


3.11 


35 


3.30 
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Table 7 
Cumulative Total Completers 
Students Who Attended - Fall 1992 





Convicted Survey 


Did Not Compete Survey 






Completers 




Completers 




n 


Cumulative 
Total 


Rate(%) 


n 


Cumulative 
Total 


Rate(%) 


All Students 


1618 


361 


22% 


496 


100 


20% 


Students who took 
fewer than 6 Credits 


631 


49 


8% 


271 


17 


6% 


Students who took 6 
to 11 Credits 


580 


126 


22% 


140 


34 


24% 


Fulltime Students 


407 


186 


46% 


85 


49 


58% 






Table 8 
Cumulative Total Completers 
First-time Students Who Attended - Fall 1992 
New Mexico State University - Alamoaordo 






Completed Survey 


Did Not Compete Survey 






Completers 




Completers 




n 


Cumulative 
Total 


J Rate(%) 


n 


Cumulative 
Total 


Rate(%) 


1 All Students 


300 


40 


13% 


80 


3 


4% 


| Students who took 
H fewer than 6 Credits 


99 


5 


5% 


52 


1 


2% | 


U Students who took 6 
| to 11 Credits 


111 


9 


8% 


20 


0 


0% 1 


H Fulltime Students 


90 


26 


1 29% 


1 8 


2 


1 25% 
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SOCIAL SECURITY NUMBER • • 

I APPENDIX 

Student Questionnaires 

What is your MOST IMPORTANT reason for attending NMSU-A? 34 
Mark only one that beil deterifett your main re iioil 

□ I want to improve my job skills or get new job skills. 

□ I want to obtain a certificate or degree from NMSU- Alamogordo. 
I want to take courses that transfer to another college. 
I enjoy taking courses for personal enrichment 

Are you currently employed? \ YES ^ NO 
If so, about how many hours per week do you work? ]0jj£U^> 
Do you plan on seeking work this semester? t3 YES □ NO 

If so, about how many hours per week would you like to work? 

Check ALL semesters that you plan to be at NMSU-A. 
ft Fall '92 □ PaU'93 □ Fall '94 

-fi Spring '93 □ Spring '94 □ Spring '95 O Beyond Summer '95 
^ Summer '93 □ Summer '94 □ Summer '95 

NMSU-A receives federal funds to help support students with special needs. If 
you believe that you fall into any of the categories listed below, please mark here and 
discuss your needs with a counselor. 

□ Person will) Disabilities 

□ Displaced Homcmnker 

□ Single Parent 
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1 
2 
3 

4 

5 

6 
7 
8 
9 



As you pursue your education at NMSU-A, do you plan to complete any specific 

program of study? J 

O YES 8K^o 

If so, please check which ones. M«r* «i awy 1 



1 Villi, 



D Emergency Medical Technician Courses 

D FAA Aircraft Mechnric Certification Courses 

D Animii Care Handli r Courses 

D Microcomputer Api-ticaliens Certificate 

D Microcomputer Ted. .lolngy Certificate 

D Medical Office Administration Certificate 

D Nurse Assistant Ccrificate 

D Office Skills CettinoMe 

D 1'hotogrsphic Studies Certificate 



10 .O Associate of Arti 

11 D Criminal Justice Assodale Degree 

12 □ Education Paraprofessional Associate Degree 

13 □ Electronics Technology Associate Degree 

14 D Fire Science Assodate Degree 

15 D Medical Lab Technology Associate Degree 

16 D Microcomputer TecNoJogy Assodate Degree 

17 D Nursing Associate Dtgrc* 

18 D Occupational Business Associate Degree 

19 □ Pre-Businces Assodate Degree 

20 D Secretarial Administration Assodata Degree 

21 O Undesignated . Associate Degree 



Do you plan to transfer to another college? 

□ To NMSU • Las Cruces 

□ To Community College of the Air Force (CCAF) 

O To Another 2-Year School 

O To Another 4-Year School 
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